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Abstract 

As data about events and objects become more commonly available, analyzing and 
understanding of interrelated temporal and spatial information is increasingly a concern for 
military commanders, intelligence analysts and business analysts. A system and method is 
5 provided for creating a multidimensional visual representation of a group of data elements 
having integrated temporal and spatial properties. The data elements are included in the visual 
representation as corresponding visual elements, such that the data elements of the group linked 
by at least one association. The system includes a visualization manager for assembling the 
group of data elements using the at least one association and for assigning a connection visual 

10 element in the visual representation between a first visual element representing a first data 
element of the group and a second visual element representing a second data element of the 
group. The system also has a spatial visualization component, such as a sprite, configured for 
generating a spatial domain of the visual representation to include a reference surface for 
providing a spatial reference frame having at least two spatial dimensions. The reference surface 

IS is for relating the first visual element to a first location of interest in the spatial reference frame 
and for relating the second visual element to a second location of interest in the spatial reference 
frame. The system also has a temporal visualization component, such as a sprite, configured for 
generating a temporal domain of the visual representation operatively coupled to the spatial 
domain, the temporal domain for providing a common temporal reference frame for the locations 

20 of interest. The temporal domain includes a first time track, such as a timeline, coupled to the 
first location of interest and a second time track coupled to the second location of interest, such 
that the first visual element is positioned on die first time track and the second visual element is 
positioned on the second time track. Each of the time tracks configured for visually representing 
a respective temporal sequence of a plurality of the data elements at each of the locations of 

25 interest of the reference surface. In implementation of the method, the connection visual element 
represents a distributed association in at least one of die domains between the first visual element 
and the second visual element such that the visual representation is displayed on a user interface 
for subsequent interaction with user events, including animation of the visual elements to help in 
the analysis of die data contained in the visual representation. 

30 
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SYSTEM AND METHOD FOR VISUALIZING CONNECTED TEMPORAL AND 
SPATIAL INFORMATION AS AN INTEGRATED VISUAL REPRESENTATION ON A 

USER INTERFACE 



5 Background of the Invention 

The present invention relates to an interactive visual presentation of multidimensional 
data on a user interface. 



Tracking and analyzing entities and streams of events, has traditionally been the domain 
1 0 of investigators, whether that be national intelligence analysts, police services or military 
intelligence. Business users also analyze events in time and location to better understand 
phenomenon such as customer behavior or transportation patterns. As data about events and 
objects become more commonly available, analyzing and understanding of interrelated temporal 
and spatial information is increasingly a concern for military commanders, intelligence analysts 
1 5 and business analysts. Localized cultures, characters, organizations and their behaviors play an 
important part in planning and mission execution. In situations of asymmetric warfare and 
peacekeeping, tracking relatively small and seemingly unconnected events over time becomes a 
means for tracking enemy behavior. For business applications, tracking of production process 
characteristics can be a means for improving plant operations. A generalized method to capture 
20 and visualize this information over time for use by business and military applications, among 
others, is needed 



Many visualization techniques and products for analyzing complex event interactions 
only display information along a single dimension, typically one of time, geography or a network 

25 connectivity diagram. Each of these types of visualizations is common and well understood. For 
example a Time-focused scheduling chart such as Microsoft (MS) Project displays various 
project events over the single dimension of time, and a Geographic Information System (GIS) 
product, such as MS MapPoint, or ESRI Arc View, is good for showing events in the single 
dimension of locations on a map. There are also link analysis tools, such as Netmap 

30 fwww.netmfloanalvtics.com^ or Visual Analytics (www.visualanalytics.com) that display events 
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as a network diagram, or graph, of objects and connections between objects. Some of these 
systems are capable of using animation to display another dimension, typically time. Time is 
played back, or scrolled, and the related spatial image display changes to reflect die state of 
information at a moment in time. However this technique relies on limited human short term 

5 memory to track and then retain temporal changes and patterns in the spatial domain. Another 
visualization technique called "small multiples" uses repeated frames of a condition or chart, 
each capturing an increment moment in time, much like looking at sequence of frames from a 
film laid side by side. Each image must be interpreted separately, and side-by-side comparisons 
made, to detect differences. This technique is expensive in terms of visual space since an image 

10 must be generated for each moment of interest, which can be problematic when trying to 
simultaneously display multiple images of adequate size that contain complex data content. 

A technique has been developed, as described in Interactive Visualization of 
Spatiotemporal Patterns using Spirals on a Geographical Map - by Hewagamage et al. that uses 

IS spiral shaped ribbons as timelines to show isolated sequences of events that have occurred at 
discrete locations on a geographical map. This technique is limited because it uses spiral 
timelines exclusively to show the periodic quality of certain types of events, while does not show 
connectivity between the temporal and spatial information of data objects at multi-locations 
within the spatial domain. Further, event data objects placed on the spirals can suffer from 

20 occlusion, thereby providing for only a limited number of events and locations viewable with die 
spiral timelines. 

It is an object of the present invention to provide a system and method for the integrated, 
interactive visual representation of a plurality of events and objects with spatial and temporal 
25 properties to obviate or mitigate at least some of the above-mentioned disadvantages. 

Summary of the Invention 

Tracking and analyzing entities and streams of events, has traditionally been the domain 
of investigators, whether that be national intelligence analysts, police services or military 
30 intelligence. Business users also analyze events in time and location to better understand 

phenomenon such as customer behavior or transportation patterns. As data about events and 



2 



CA 02461118 2004-03-15 



objects become more commonly available, analyzing and understanding of interrelated temporal 
and spatial information is increasingly a concern for military commanders, intelligence analysts 
and business analysts. Contrary to present analysis tools, a system and method is provided for 
creating a multidimensional visual representation of a group of data elements having integrated 
5 temporal and spatial properties. The data elements are included in the visual representation as 
corresponding visual elements, such that the data elements of the group linked by at least one 
association. The system includes a visualization manager for assembling the group of data 
elements using the at least one association and for assigning a connection visual element in the 
visual representation between a first visual element representing a first data element of the group 

10 and a second visual element representing a second data element of the group. The system also 
has a spatial visualization component, such as a sprite, configured for generating a spatial 
domain of die visual representation to include a reference surface for providing a spatial 
reference frame having at least two spatial dimensions. The reference surface is for relating the 
first visual element to a first location of interest in the spatial reference frame and for relating the 

15 second visual element to a second location of interest in the spatial reference frame. The system 
also has a temporal visualization component, such as a sprite, configured for generating a 
temporal domain of the visual representation operatively coupled to the spatial domain, the 
temporal domain for providing a common temporal reference frame for the locations of interest 
The temporal domain includes a first time track, such as a timeline, coupled to the first location 

20 of interest and a second time track coupled to the second location of interest, such that the first 
visual element is positioned on the first time track and the second visual element is positioned on 
the second time track. Each of the time tracks configured for visually representing a respective 
temporal sequence of a plurality of the data elements at each of the locations of interest of the 
reference surface. In implementation of the method, the connection visual element represents a 

25 distributed association in at least one of the domains between the first visual element and the 
second visual element such that the visual representation is displayed on a user interface for 
subsequent interaction with user events, including animation of the visual elements to help in die 
analysis of the data contained in the visual representation. 

30 
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According to the present invention there is provided a method for creating a 
multidimensional visual representation of a group of data elements having integrated temporal 
and spatial properties, the data elements being included in the visual representation as 
corresponding visual elements, the data elements of the group linked by at least one association, 
5 the method comprising the steps of: assembling the group of data elements using the at least one 
association; generating a spatial domain of the visual representation to include a reference 
surface for providing a spatial reference frame having at least two spatial dimensions, the 
reference surface for relating a first visual element representing a first data element of the group 
to a first location of interest in the spatial reference frame and relating a second visual element 

10 representing a second data element of the group to a second location of interest in the spatial 
reference frame; generating a temporal domain of the visual representation operatively coupled 
to the spatial domain, the temporal domain for providing a common temporal reference frame for 
the locations of interest, the temporal domain including a first time track coupled to the first 
location of interest and a second time track coupled to the second location of interest, the first 

1 5 visual element positioned on the first time track and the second visual element positioned on the 
second time track, each of die time tracks configured for visually representing a respective 
temporal sequence of a plurality of the data elements at each of the locations of interest of the 
reference surface; and assigning a connection visual element in the visual representation between 
the first visual element and die second visual element, the connection visual element for 

20 representing a distributed association in at least one of the domains between the first visual 
element and the second visual element; wherein the visual representation is displayed on a user 
interface for subsequent interaction with user events. 

According to a further aspect of the present invention there is provided a system for 
25 creating a multidimensional visual representation of a group of data elements having integrated 
temporal and spatial properties, the data elements being included in die visual representation as 
corresponding visual elements, the data elements of the group linked by at least one association, 
the system comprising: a visualization manager for assembling the group of data elements using 
the at least one association and for assigning a connection visual element in die visual 
30 representation between a first visual element representing a first data element of the group and a 
second visual element representing a second data element of the group; a spatial visualization 
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component configured for generating a spatial domain of the visual representation to include a 
reference surface for providing a spatial reference frame having at least two spatial dimensions, 
the reference surface for relating the first visual element to a first location of interest in the 
spatial reference frame and relating the second visual element to a second location of interest in 
5 the spatial reference frame; and a temporal visualization component configured for generating a 
temporal domain of the visual representation operatively coupled to the spatial domain, the 
temporal domain for providing a common temporal reference frame for die locations of interest, 
the temporal domain including a first time track coupled to the first location of interest and a 
second time track coupled to the second location of interest, the first visual element positioned 

10 on the first time track and the second visual element positioned on the second time track, each of 
the time tracks configured for visually representing a respective temporal sequence of a plurality 
of the data elements at each of the locations of interest of the reference surface; and wherein the 
connection visual element represents a distributed association in at least one of the domains 
between the first visual element and the second visual element such that the visual representation 

IS is displayed on a user interface for subsequent interaction with user events. 

According to a still further aspect of the present invention there is provided a computer 
program product for creating a multidimensional visual representation of a group of data 
elements having integrated temporal and spatial properties, the data elements being included in 

20 the visual representation as corresponding visual elements, the data elements of the group linked 
by at least one association, the computer program product comprising: a computer readable 
medium; a visualization module stored on the computer readable medium for assembling the 
group of data elements using the at least one association and for assigning a connection visual 
element in the visual representation between a first visual element representing a first data 

25 element of the group and a second visual element representing a second data element of the 
group; a spatial visualization module stored on the computer readable medium for generating a 
spatial domain of the visual representation to include a reference surface for providing a spatial 
reference frame having at least two spatial dimensions, the reference surfece for relating the first 
visual element to a first location of interest in the spatial reference frame and relating the second 

30 visual element to a second location ofinterest in the spatial reference frame; and a temporal 

visualization module stored on the computer readable medium for generating a temporal domain 
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of the visual representation operatively coupled to the spatial domain, the temporal domain for 
providing a common temporal reference frame for the locations of interest, the temporal domain 
including a first time track coupled to the first location of interest and a second time track 
coupled to the second location of interest, the first visual element positioned on the first time 

5 track and the second visual element positioned on the second time track, each of the time tracks 
configured for visually representing a respective temporal sequence of a plurality of the data 
elements at each of the locations of interest of the reference surface; wherein the connection 
visual element represents a distributed association in at least one of the domains between the first 
visual element and the second visual element such that the visual representation is displayed on a 

10 user interface for subsequent interaction with user events. 



Brief Description of the Drawings 

A better understanding of these and other embodiments of the present invention can be 
obtained with reference to the following drawings and detailed description of the preferred 
1 5 embodiments, in which: 

Figure 1 is a block diagram of a data processing system for a visualization tool; 

Figure 2 shows further details of the data processing system of Figure 1; 

Figure 3 shows further details of the visualization tool of Figure 1 ; 

Figure 4 shows further details of a visualization representation for display on a 
20 visualization interface of the system of Figure 1 ; 

Figure 5 is an example visualization representation of Figure 1 showing Events in 
Concurrent Time and Space; 

Figure 6 shows example data objects and associations of Figure 1; 

Figure 7 shows further example data objects and associations of Figure 1 ; 
25 Figure 8 shows changes in orientation of a reference surface of the visualization 

representation of Figure 1; 

Figure 9 is an example timeline of Figure 8; 

Figure 10 is a further example timeline of Figure 8; 

Figure 1 1 is a further example timeline of Figure 8 showing a time chart; 
30 Figure 12 is a further example of the time chart of Figure 1 1; 
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Figure 13 shows example user controls for the visualization representation of Figure 5; 

Figure 14 shows an example operation of the tool of Figure 3; 

Figure IS shows a further example operation of the tool of Figure 3; 

Figure 16 shows a further example operation of the tool of Figure 3; 
5 Figure 1 7 shows an example visualization representation of Figure 4 containing events 

and target tracking over space and time showing connections between events; 

Figure 18 shows an example visualization representation containing events and target 
tracking over space and time showing connections between events on a time chart of Figure 1 1 , 
and 

10 Figure 19 is an example operation of the visualization tool of Figure 3. 

It is noted that similar references are used in different figures to denote similar 
components. 

Detailed Description of the Preferred Embodiment 

1 5 The following detailed description of the embodiments of the present invention does not 

limit the implementation of the invention to any particular computer programming language. 
The present invention may be implemented in any computer programming language provided 
that the OS (Operating System) provides the facilities that may support the requirements of the 
present invention. A preferred embodiment is implemented in the Java computer programming 

20 language (or other computer programming languages in conjunction with C/C++). Any 
limitations presented would be a result of a particular type of operating system, computer 
programming language, or data processing system and would not be a limitation of the present 
invention. 

25 Visualization Environment 

Referring to Figure 1, a visualization data processing system 100 includes a visualization 
tool 12 for processing a collection of data objects 14 as input data elements to a user interface 
202. The data objects 14 are combined with a respective set of associations 16 by the tool 12 to 
generate an interactive visual representation 1 8 on the visual interface (VI) 202. The data objects 
30 14 include event objects 20, location objects 22, and entity objects 24, as further described 
below. The set of associations 16 include individual associations 26 that associate together 
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various subsets of the objects 20, 22, 24, as further described below. Management of the data 
objects 14 and set of associations 16 are driven by user events 109 of a user (not shown) during 
interaction with the visual representation 18. 

5 Data processing system 

Referring to Figure 2, the data processing system 1 00 has a user interface 1 08 for 
interacting with the tool 12, the user interface 108 being connected to a memory 102 via a BUS 
106. The interface 108 is coupled to a processor 104 via the BUS 106, to interact with user 
events 109 to monitor or otherwise instruct the operation of the tool 12 via an operating system 

10 1 10. The user interface 108 can include one or more user input devices such as but not limited to 
a QWERTY keyboard, a keypad, a trackwheel, a stylus, a mouse, and a microphone. The visual 
interface 202 is considered the user output device, such as but not limited to a computer screen 
display. If the screen is touch sensitive, then the display can also be used as the user input device 
as controlled by the processor 104. Further, it is recognized that the data processing system 100 

15 can include a computer readable storage medium 46 coupled to the processor 104 for providing 
instructions to the processor 104 and/or the tool 12. The computer readable medium 46 can 
include hardware and/or software such as, by way of example only, magnetic disks, magnetic 
tape, optically readable medium such as CD/DVD ROMS, and memory cards. In each case, the 
computer readable medium 46 may take the form of a small disk, floppy diskette, cassette, hard 

20 disk drive, solid-state memory card, or RAM provided in the memory 1 02. It should be noted 
that the above listed example computer readable mediums 46 can be used either alone or in 
combination. 



Referring again to Figure 2, the tool 12 interacts via link 116 with a VI manager 112 (also 
25 known as a visualization tenderer) of the system 1 00 for presenting the visual representation 18 
on the visual interface 202. The tool 12 also interacts via link 118 with a data manager 114 of 
the system 100 to coordinate management of the data objects 14 and association set 16 from data 
files or tables 122 of the memory 102. It is recognized that the objects 14 and association set 16 
could be stored in the same or separate tables 122, as desired. The data manager 114 can receive 
30 requests for storing, retrieving, amending, or creating the objects 14 and association set 16 via 
the tool 12 and/or directly via link 120 from the VI manager 1 12, as driven by the user events 
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109 and/or independent operation of the tool 12. The data manager 1 14 manages die objects 14 
and association set 16 via link 123 with the tables 122. Accordingly, the tool 12 and managers 
1 12, 1 14 coordinate the processing of data objects 14, association set 16 and user events 109 
with respect to the content of the screen representation 18 displayed in the visual interface 202. 

5 

Too) Information Model 

Referring to Figure 1 , a tool information model is composed of die four basic data 
elements (objects 20, 22, 24 and associations 26) that can have corresponding display elements 
in the visual representation 18. The four elements are used by the tool 12 to describe 
1 0 interconnected activities and information in time and space as the integrated visual representation 
18, as further described below. 

Event data objects 20 

Events are data objects 20 that represent any action that can be described The following 
1 5 are examples of events; 

- Bill was at Toms house at 3pm, 

- Tom phoned Bill on Thursday, 

- A tree fell m the forest at 4:13 am, June 3, 1993 and 

- Tom will move to Spain in the summer of 2004. 

20 

The Event is related to a location and a time at which the action took place, as well as several 
data properties and display properties including such as but not limited to; a short text label, 
description, location, start-time, end-time, general event type, icon reference, visual layer 
settings, priority, status, user comment, certainty value, source of information, and default + 
25 user-set color. The event data object 20 can also reference files such as images or word 
documents. 

Locations and times may be described with varying precision. For example, event times 
can be described as "during the week of January 5*" or "in the month of September". Locations 
30 can be described as "Spain" or as "New York" or as a specific latitude and longitude. 

Entity data nhjppts 24 
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Entities are data obj ects 24 that represent any thing related to or involved in an event, 
including such as but not limited to; people, objects, organizations, equipment, businesses, 
observers, affiliations etc. Data included as part of die Entity data object 24 can be short text 
label, description, general entity type, icon reference, visual layer settings, priority, status, user 

5 comment, certainty value, source of information, and default + user-set color. The entity data can 
also reference files such as images or word documents. It is recognized in reference to Figures 6 
and 7 that the term Entities includes 'Teople", as well as equipment (e.g. vehicles), an entire 
organization (e.g. corporate entity), currency, and any other object that can be tracked for 
movement in the spatial domain 400. It is also recognized that the entities 24 could be stationary 

10 objects such as but not limited to buildings. Further, entities can be phone numbers and web 
sites. To be explicit, the entities 24 as given above by example only can be regarded as Actors 

Locations data objects 22 

IS Locations are data objects 22 that represent a place within a spatial context/domain, such 

as a geospatial map, a node in a diagram such as a flowchart, or even a conceptual place such as 
"Shang-ri-la" or other "locations" that cannot be placed at a specific physical location on a map 
or other spatial domain. Each Location data object 22 can store such as but not limited to; 
position coordinates, a label, description, color information, precision information, location type, 

20 non-geo spatial flag and user comments. 

Associations 

Event 20, Location 22 and Entity 24 are combined into groups or subsets of the data 
objects 14 in the memory 102 (see Figure 2) using associations 26 to describe real-world 

25 occurrences. The association is defined as an information object that describes a pairing between 
2 data objects 1 4. For example, in order to show that a particular entity was present when an 
event occurred, the corresponding association 26 is created to represent that Entity X **was 
presort at" Event A. For example, associations 26 can include such as but not limited to; 
describing a communication connection between two entities 24, describing a physical 

30 movement connection between two locations of an entity 24, and a relationship connection 
between a pair of entities 24 (e.g. family related and/or organizational related). It is recognised 
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that the associations 26 can describe direct and indirect connections. Other examples can include 
phone numbers and web sites. 

Visualization Tool 12 

5 Referring to Figure 3, the visualization tool 12 has a visualization manager 300 for 

interacting with the data obj ects 14 for presentation to die interface 202 via the VI manager 112. 
The Data Objects 14 are formed into groups 27 through the associations 26 and processed by the 
Visualization Manager 300. The groups 27 comprise selected subsets of the objects 20, 22, 24 
combined via selected associations 26. This combination of data objects 14 and association sets 

10 16 can be accomplished through predefined groups 27 added to the tables 122 and/or through the 
user events 109 during interaction of the user directly with selected data objects 14 and 
association sets 1 6 via the controls 306. It is recognized that the predefined groups 27 could be 
loaded into die memory 102 (and tables 122) via the computer readable medium 46 (see Figure 
2). The Visualization manager 300 also processes user event 109 input through interaction with 

1 5 a time slider and other controls 306, including several interactive controls for supporting 
navigation and analysis of information within the visual representation 18 (see Figure 1) as 
further described below. 

The Visualization Manager 300 processes the translation from raw data objects 1 4 to the 
20 visual representation 18. First, Data Objects 14 and associations 16 are formed by the 

Visualization Manager 300 into the groups 27, as noted in the tables 122, and then processed 
The Visualization Manager 300 matches the raw data objects 14 and associations 1 6 with sprites 
308 (i.e. visual processing objects/components that know how to draw and render visual 
elements for specified data objects 14 and associations 16) and sets a drawing sequence for 
25 implementation by the VI manager 1 1 2. The sprites 308 are visualization components that take 
predetermined information schema as input and output graphical elements such as lines, text, 
images and icons to the computers graphics system. Entity 24, event 20 and location 22 data 
objects each can have a specialized sprite 308 type designed to represent diem. A new sprite 
instance is created for each entity, event and location instance to manage their representation in 
30 the visual representation 18 on the display. 
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The sprites 308 are processed in order by the visualization manager 300, starting with the 
spatial domain (terrain) context and locations, Mowed by Events and Timelines, and finally 
Entities. Timelines are generated and Events positioned along them. Entities are rendered last by 
the sprites 308 since the entities depend on Event positions. It is recognised that processing 
5 order of the sprites 308 can be other than as described above. 

The Visualization manager 112 renders the sprites 308 to create the final image including 
visual elements representing the data objects 14 and associates 16 of the groups 27, for display as 
10 the visual representation 18 on the interface 202. After the visual representation 18 is on die 
interface 202, the user event 1 09 inputs flow into the Visualization Manager, through the VI 
manager 1 12 and cause the visual representation 18 to be updated. The Visualization Manager 
300 can be optimized to update only those sprites 308 that have changed in order to maximize 
interactive performance between the user and the interface 202. 

15 

Layout of the Visualization Representation 18 

The visualization technique of the visualization tool 12 is designed to improve perception 
of entity activities, movements and relationships as they change over time in a concurrent time- 

20 geographic or time-diagrammatical context. The visual representation 18 of the data objects 14 
and associations 16 consists of a combined temporal-spatial display to show interconnecting 
streams of events over a range of time on a map or other schematic diagram space, both hereafter 
referred to in common as a spatial domain 400 (see Figure 4). Events can be represented within 
an X,Y,T coordinate space, in which die X,Y plane shows the spatial domain 400 (e.g. 

25 geographic space) and the Z-axis represents a time series into the future and past, referred to as a 
temporal domain 402. In addition to providing the spatial context, a reference surface (or 
reference spatial domain) 404 marks an instant of focus between before and after, such that 
events "occur" when they meet the surface of the ground reference surface 404. Figure 4 shows 
how the visualization manager 300 (see Figure 3) combines individual frames 406 (spatial 

30 domains 400 taken at different times Ti 407) of event/entity/location visual elements 410, which 
are translated into a continuous integrated spatial and temporal visual representation 18. It 
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should be noted connection visual elements 412 can represent presumed location (interpolated) 
of Entity between the discrete event/entity/location represented by the visual elements 410. 
Another interpretation for connections elements 412 could be signifying communications 
between different Entities at different locations, which are related to the same event as further 
5 described below. 

Referring to Figure 5, an example visual representation 1 8 visually depicts events over 
time and space in an x, y, t space (or x, y, z, t space with elevation data). The example visual 
representation 18 generated by the tool 12 (see Figure 2) is shown having the time domain 402 as 

10 days in April, and the spatial domain 400 as a geographical map providing the instant of focus 
(of the reference surface 404) as sometime around noon on April 23 - the intersection point 
between the timelines 422 and the reference surface 404 represents the instant of focus. The 
visualization representation 18 represents the temporal 402, spatial 400 and connectivity 
elements 412 (between two visual elements 410) of information within a single integrated picture 

IS on the interface 202 (see Figure 1). Further, the tool 12 provides an interactive analysis tool for 
the user with interface controls 306 to navigate the temporal, spatial and connectivity 
dimensions. The tool 12 is suited to the interpretation of any information in which time, location 
and connectivity are key dimensions that are interpreted together. The visual representation 18 is 
used as a visualization technique for displaying and tracking events, people, and equipment 

20 within the combined temporal and spatial domains 402, 400 display. The visual representation 
1 8 can be applied as an analyst workspace for exploration, deep analysis and presentation for 
such as but not limited to: 

- Situations involving people and organizations that interact over time and in which 
geography or territory plays a role; 

25 - Storing and reviewing activity reports over a given period. Used in this way the 

representation 18 could provide a means to determine a living history, context and 
lessons learned from past events; and 

- As an analysis and presentation tool for long term tracking and surveillance of persons 
and equipment activities. 

30 



13 



CA 02461118 2004-03-15 



The visualization tool 12 provides the visualization representation 18 as an interactive 
display, such that the users (e.g. intelligence analysts, business marketing analysts) can view, and 
work with, large numbers of events. Further, perceived patterns, anomalies and connections can 
be explored and subsets of events can be grouped into "story" or hypothesis fragments* The 
visualization tool 12 includes a variety of capabilities such as but not limited to: 

■ An event-based information architecture with places, events, entities (e.g. people) and 
relationships; 

Past and future time visibility and animation controls; 

Data input wizards for describing single events and for loading many events from a table; 
10 ■ Entity and event connectivity analysis in time and geography; 
Path displays in time and geography; 

Configurable workspaces allowing ad hoc, drag and drop arrangements of events; 
Search, filter and drill down tools; 

Creation of sub-groups and overlays by selecting events and dragging them into sets 
15 (along with associated spatial/time scope properties); and 

1 Adaptable display functions including dynamic show / hide controls. 

Example objects 14 with associations 16 

In fixe visualization tool 12, specific combinations of associated data elements (objects 
20 20, 22, 24 and associations 26) can be defined. These defined groups 27 are represented visually 
as visual elements 410 in specific ways to express various types of occurrences in the visual 
representation 18. The following are examples of how the groups 27 of associated data elements 
can be formed to express specific occurrences and relationships shown as the connection visual 
elements 412. 

25 

Referring to Figures 6 and 7, example groups 27 (denoting common real world 
occurrences) are shown with selected subsets of the objects 20, 22, 24 combined via selected 
associations 26. The corresponding visualization representation 18 is shown as well including 
the temporal domain 402, the spatial domain 400, connection visual elements 412 and the visual 
30 elements 410 representing the event/entity/location combinations. It is noted that example 
applications of the groups 27 are such as but not limited to those shown in Figures 6 and 7. In 
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the Figures 6 and 7 it is noted that event objects 20 are labeled as "Event 1", "Event 2", location 
objects 22 are labeled as "Location A", "Location B", and entity objects 24 are labeled as "Entity 
X", "Entity Y". The set of associations 16 are labeled as individual associations 26 with 
connections labeled as either solid or dotted lines 412 between two events, or dotted in the case 
5 of an indirect connection between two locations. 

Visual Elements Corresponding to Spatial and Temporal Domains 

The visual elements 410 and 412, their variations and behavior facilitate interpretation of 
the concurrent display of events in the time 402 and space 400 domains. In general, events 
10 reference the location at which they occur and a list of Entities and their role in the event The 
time at which the event occurred or the time span over which the event occurred are stored as 
parameters of the event. 

Spatial Domain Representation 

1 5 Referring to Figure 8, the primary organizing element of the visualization representation 

18 is the 2D/3D spatial reference frame (subsequently included herein with reference to the 
spatial domain 400). The spatial domain 400 consists of a true 2D/3D graphics reference surface 
404 in which a 2D or 3 dimensional representation of an area is shown. This spatial domain 400 
can be manipulated using a pointer device (not shown -part of the controls 306 - see Figure 3) 

20 by the user of the interface 108 (see Figure 2) to rotate the reference surface 404 with respect to a 
viewpoint 420 or viewing ray extending from a viewer 423. The user (i.e. viewer 423) can also 
navigate the reference surface 404 by scrolling in any direction, zooming in or out of an area and 
selecting specific areas of focus. In this way the user can specify the spatial dimensions of an 
area of interest the reference surface 404 in which to view events in time. The spatial domain 

25 400 represents space essentially as a plane (e.g. reference surface 404), however is capable of 
representing 3 dimensional relief within that plane in order to express geographical features 
involving elevation. The spatial domain 400 can be made transparent so that timelines 422 of the 
temporal domain 402 can extend behind the reference surface 404 are still visible to the user. 
Figure 8 shows how the viewer 423 facing timelines 422 can rotate to face the viewpoint 420 no 

30 matter how the reference surface 404 is rotated in 3 dimensions with respect to the viewpoint 
420. 
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Hie spatial domain 400 includes visual elements 410, 412 (see Figure 4) that can 
represent such as but not limited to map information, digital elevation data, diagrams, and 
images used as the spatial context. These types of spaces can also be combined into a workspace. 
5 The user can also create diagrams using drawing tools (of the controls 306 - see Figure 3) 
provided by the visualization tool 12 to create custom diagrams and annotations within the 
spatial domain 400. 

10 Event Representation and Interactions 

Referring to Figures 4 and 8, events are represented by a glyph, or icon as the visual 
element 410, placed along the timeline 422 at the point in time that the event occurred. The 
glyph can be actually a group of graphical objects, or layers, each of which expresses the content 
of the event data object 20 (see Figure 1) in a different way. Each layer can be toggled and 

1 5 adjusted by the user on a per event basis, in groups or across all event instances. The graphical 
objects or layers for event visual elements 410 are such as but not limited to: 

1. Text label 

The Text label is a text graphic meant to contain a short description of the event content 
20 This text always feces the viewer 423 no matter how the reference surface 404 is 

oriented. The text label incorporates a de-cluttering function that separates it from other 
labels if they overlap. When two events are connected with a line (see connections 412 
below) the label will be positioned at the midpoint of die connection line between the 
events. The label will be positioned at the end of a connection line that is clipped at the 
25 edge of the display area. 

2. Indicator - Cylinder, Cube or Sphere 

The indicator marks the position in time. The color of die indicator can be manually set 
by the user in an event properties dialog. Color of event can also be set to match the 
30 Entity that is associated with it. The shape of the event can be changed to represent 

different aspect of information and can be set by the user. Typically it is used to represent 
a dimension such as type of event or level of importance. 
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3. Icon 

An icon or image can also be displayed at the event location. This icon may used to 
describe some aspect of the content of the event. This icon may be user-specified or 
5 entered as part of a data file of the tables 122 (see Figure 2). 

4. Connection elements 412 

Connection elements 412 can be lines, or other geometrical curves, which are solid or 
dashed lines that show connections from an event to another event, place or target A 
10 connection element 412 may have a pointer or arrowhead at one end to indicate a 

direction of movement, polarity, sequence or other vector-like property. If the connected 
object is outside of the display area, the connection element 412 can be coupled at the 
edge of the reference surface 404 and the event label will be positioned at the clipped end 
of the connection element 412. 

15 

5. Time Range Indicator 

A Time Range Indicator (not shown) appears if an event occurs over a range of time. Hie 
time range can be shown as a line parallel to the timeline 422 with ticks at the end points. 
The event Indicator (see above) preferably always appears at the start time of the event. 

20 

The Event visual element 410 can also be sensitive to interaction. The following user 
events 109 via the user interface 108 (see Figure 2) are possible, such as but not limited to: 

Mouse-Left-Click: 

25 Selects the visual element 410 of the visualization representation 18 on die VI 202 (see 

Figure 2) and highlights it, as well as simultaneously deselecting any previously selected 
visual element 410, as desired 

Ctri-Mouse-Left-Click and Shift-Mouse-Left-Click 
30 Adds the visual element 410 to an existing selection set. 
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Mouse-Left-Double-Click: 

Opens a file specified in an event data parameter if it exists. The file will be opened in a 
system-specified default application window on the interface 202 based on its file type. 

5 Mouse-Right-Click: 

Displays an in-context popup menu with options to hide, delete and set properties. 



Mouse over Drilldown: 

When the mouse pointer (not shown) is placed over the indicator, a text window is 
10 displayed next to the pointer, showing information about the visual element 410. When 

the mouse pointer is moved away from the indicator, the text window disappears. 

Location Repre^eptpjion 

Locations are visual elements 410 represented by a glyph, or icon, placed on the 
1 5 reference surface 404 at the position specified by the coordinates in the corresponding location 
data object 22 (see Figure 1). The glyph can be a group of graphical objects, or layers, each of 
which expresses the content of the location data object 22 in a different way. Each layer can be 
toggled and adjusted by the user on a per Location basis, in groups or across all instances. The 
visual elements 410 (e.g. graphical objects or layers) for Locations are such as but not limited to: 

20 

1. Text Label 

The Text label is a graphic object for displaying the name of the location. This text 
always feces the viewer 422 no matter how the reference surface 404 is oriented. The text 
label incorporates a de-cluttering function that separates it from other labels if they 
25 overlap. 

2. Indicator 

The indicator is an outlined shape that marks the position or approximate position of the 
Location data obj ect 22 on the reference surface 404 . There are, such as but not limited 
30 to, 7 shapes that can be selected for the locations visual elements 410 (marker) and the 

shape can be filled or empty. The outline thickness can also be adjusted. The default 
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setting can be a circle and can indicate spatial precision with size. For example, more 
precise locations, such as addresses, are smaller and have thicker line width, whereas a 
less precise location is larger in diameter, but uses a thin line width. 

5 The Location visual elements 410 are also sensitive to interaction. The following 

interactions are possible: 

Mouse-Left-Click: 

Selects the location visual element 410 and highlights it, while deselecting any previously 
10 selected location visual elements 410. 

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click 

Adds the location visual element 410 to an existing selection set 

15 Mouse-Uft-Double-Click: 

Opens a file specified in a Location data parameter if it exists. The file will be opened in 
a system-specified default application window based on its file type. 

Mouse-Right-Click: 

20 Displays an in-context popup menu with options to hide, delete and set properties of the 

location visual element 410. 

Mouseover Drilldown: 

When the Mouse pointer is placed over the location indicator, a text window showing 
25 information about the location visual element 410 is displayed next to the pointer. When 

the mouse pointer is moved away from the indicator, the text window disappears. 

Mouse-Left-Ctick-Hold-and-Drag: 

Interactively repositions the location visual element 410 by dragging it across the 
30 reference surface 404. 
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Nop-Spatfrl tacrtiora 

Locations have the ability to represent indeterminate position. These are referred to as 
non-spatial locations. Locations tagged as non-spatial can be displayed at the edge of the 
reference surface 404 just outside of the spatial context of the spatial domain 400. These non- 
5 spatial or virtual locations can be always visible no matter where the user is currently zoomed in 
on the reference surface 404. Events and Timelines 422 that are associated with non-spatial 
Locations can be rendered the same way as Events with spatial Locations. 

Entity Representation 

10 Entity visual elements 410 are represented by a glyph, or icon, and can be positioned on 

the reference surface 404 or other area of die spatial domain 400, based on associated Event data 
that specifies its position at the current Moment of Interest 900 (see Figure 9) (i.e. specific point 
on the timeline 422 that intersects the reference surface 404). If the current Moment of Interest 
900 lies between 2 events in time that specify different positions, the Entity position will be 

1 5 interpolated between the 2 positions. Alternatively, the Entity could be positioned at the most 
recent known location on he reference surface 404. The Entity glyph is actually a group of the 
entity visual elements 410 (e.g. graphical objects, or layers) each of which expresses the content 
of the event data object 20 in a different way. Each layer can be toggled and adjusted by the user 
on a per event basis, in groups or across all event instances. The entity visual elements 410 are 

20 such as but not limited to: 

1. Text Label 

The Text label is a graphic object for displaying die name of the Entity. This text always 
faces the viewer no matter how the reference surface 404 is oriented. The text label 
25 incorporates a de-cluttering function that separates it from other labels if they overlap. 

2. Indicator 

The indicator is a point showing the interpolated or real position of the Entity in die 
spatial context of the reference surface 404. The indicator assumes die color specified as 
30 an Entity color in the Entity data model. 
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3. Image Icon 

An icon or image is displayed at the Entity location. This icon may used to represent the 
identity of the Entity. The displayed image can be user-specified or entered as part of a 
data file. The Image Icon can have an outline border that assumes the color specified as 
5 the Entity color in the Entity data model. The Image Icon incorporates a de-cluttering 

function that separates it from other Entity Image Icons if they overlap. 

4. Past Trail 

The Past Trail is the connection visual element 412, as a series of connected lines that 
10 trace previous known positions of the Entity over time, starting from the current Moment 

of Interest 900 and working backwards into past time of the timeline 422. Previous 
positions are defined as Events where the Entity was known to be located The Past Trail 
can mark the path of the Entity over time and space simultaneously. 

15 5. Future Trail 

The Future Trail is the connection visual element 412, as a series of connected lines that 
trace future known positions of the Entity over time, starting from the current Moment of 
Interest 900 and working forwards into future time. Future positions are defined as 
Events where the Entity is known to be located. The Future Trail can mark the future path 

20 of die Entity over time and space simultaneously. 

The Entity representation is also sensitive to interaction. The following interactions are 
possible, such as but not limited to: 
Mouse-Left-Click: 

25 Selects the entity visual element 410 and highlights it and deselects any previously 

selected entity visual element 410. 

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click 

Adds the entity visual element 410 to an existing selection set 



30 



Mouse-Left-Double-Click: 
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Opens the file specified in an Entity data parameter if it exists. The file will be opened in 
a system-specified default application window based on its file type. 

Mouse-Right-Click: 

5 Displays an in-context popup menu with options to hide, delete and set properties of the 

entity visual element 410. 

Mouseover Drilldown: 

When the Mouse pointer is placed over the indicator, a text window showing information 
10 about the entity visual element 410 is displayed next to the pointer. When the mouse 

pointer is moved away from the indicator, the text window disappears. 

Temporal Domain including Timelines 

Referring to Figures 8 and 9, the temporal domain provides a common temporal reference 

1 S frame for the spatial domain 400, whereby the domains 400, 402 are operatively coupled to one 
another to simultaneously reflect changes in interconnected spatial and temporal properties of the 
data elements 14 and associations 16. Timelines 422 (otherwise known as time tracks) represent 
a distribution of the temporal domain 402 over the spatial domain 400, and are a primary 
organizing element of information in the visualization representation 18 that make it possible to 

20 display events across time within the single spatial display on the VI 202 (see Figure 1). 
Timelines 422 represent a stream of time through a particular Location visual element 410a 
positioned on the reference surface 404 and can be represented as a literal line in space. Other 
options for representing the timelines/time tracks 422 are such as but not limited to curved 
geometrical shapes (e.g. spirals) including 2D and 3D curves when combining two or more 

25 parameters in conjuction with the temporal dimension. Each unique Location of interest 

(represented by the location visual element 410a) has one Timeline 422 that passes through it. 
Events (represented by event visual elements 410b) that occur at that Location are arranged 
along this timeline 422 according to the exact time or range of time at which the event occurred 
In this way multiple events (represented by respective event visual elements 410b) can be 

30 arranged along die timeline 422 and the sequence made visually apparent. A single spatial view 
will have as many timelines 422 as necessary to show every Event at every location within the 
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current spatial and temporal scope, as defined in the spatial 400 and temporal 402 domains (see 
Figure 4) selected by the user. In order to make comparisons between events and sequences of 
event between locations, the time range represented by multiple timelines 422 projecting through 
the reference surface 404 at different spatial locations is synchronized. In other words the time 
S scale is the same across all timelines 422 in the time domain 402 of the visual representation 18. 
Therefore, it is recognised that the timelines 422 are used in the visual representation 18 to 
visually depict a graphical visualization of the data objects 14 over time with respect to their 
spatial properties/attributes. 

10 Representing Current, Past and Future 

Three distinct strata of time are displayed by the timelines 422, namely, 

1. The "moment of interest" 900 or browse time, as selected by the user, 

2. a range 902 of past time preceding the browse time called "past", and 

3. a range 904 of time after the moment of interest 900, called "future" 

15 

On a 3D Timeline 422, the moment of focus 900 is the point at which the timeline 
intersects the reference surface 404. An event that occurs at the moment of focus 900 will appear 
to be placed on the reference surface 404 (event representation is described above). Past and 
future time ranges 902, 904 extend on either side (above or below) of the moment of interest 900 
20 along the timeline 422. Amount of time into the past or future is proportional to the distance 

from the moment of focus 900. The scale of time may be linear or logarithmic in either direction. 
The user may select to have the direction of future to be down and past to be up or vice versa. 

\ 

There are three basic variations of Spatial Timelines 422 that emphasize spatial and 
25 temporal qualities to varying extents. Each variation has a specific orientation and 
implementation in terms of its visual construction and behavior in the visualization 
representation 1 8 (see Figure 1). The user may choose to enable any of the variations at any time 
during application runtime, as further described below. 

30 3D Z-axts Timeline 

Figure 10 shows how 3D Timelines 422 pass through reference surface 404 locations 
410a. 3D timelines 422 are locked in orientation (angle) with respect to the orientation of the 
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reference surface 404 and are affected by changes in perspective of the reference surface 404 
about the viewpoint 420 (see Figure 8). For example, the 3D Timelines 422 can be oriented 
normal to the reference surface 404 and exist within its coordinate space. Within the 3D spatial 
domain 400, the reference surface 404 is rendered in the X-Y plane and the timelines 422 run 

5 parallel to the Z-axis through locations 410a on the reference surface 404. Accordingly, the 3D 
Timelines 422 move with the reference surface 404 as it changes in response to user navigation 
commands and viewpoint changes about the viewpoint 420, much like flag posts are attached to 
the ground in real life. The 3D timelines 422 are subject to the same perspective effects as other 
objects in the 3D graphical window of the VI 202 (see Figure 1) displaying the visual 

1 0 representation 1 8. The 3D Timelines 422 can be rendered as thin cylindrical volumes and are 
rendered only between events 410a with which it shares a location and die location 410a on die 
reference surface 404. The timeline 422 may extend above the reference surface 404, below the 
reference surface 404, or both. If no events 410b for its location 410a are in view the timeline 
422 is not shown on the visualization representation 18. 

15 

3D Viewer Faqng Tfoeljn js 

Referring to Figure 8, 3D Viewer-facing Timelines 422 are similar to 3D Timelines 422 
except that they rotate about a moment of focus 425 (point at which the viewing ray of the 
viewpoint 420 intersects the reference surface 404) so that the 3D Viewer-facing Timeline 422 

20 always remain perpendicular to viewer 423 from which the scene is rendered. 3D Viewer- 
facing Timelines 422 are similar to 3D Timelines 422 except that they rotate about the moment 
of focus 425 so that they are always parallel to a plane 424 normal to the viewing ray between 
\ the viewer 423 and the moment of focus 425. The effect achieved is that the timelines 422 are 

always rendered to face the viewer 423, so that the length of the timeline 422 is always 

25 maximized and consistent. This technique allows the temporal dimension of the temporal domain 
402 to be read by the viewer 423 indifferent to how the reference surface 404 many be oriented 
to the viewer 423. This technique is also generally referred to as "billboarding" because the 
information is always oriented towards the viewer 423. Using this technique the reference 
surface 404 can be viewed from any direction (including directly above) and the temporal 

30 information of the tiiyielit^ 422 remains readable. 

Linked TimeChart Timelines 
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Referring to Figure 11, showing how an overlay time chart 430 is connected to die 
reference surface 404 locations 410a by timelines 422. The timelines 422 of the Linked 
TimeChart 430 are timelines 422 that connect the 2D chart 430 (e.g. grid) in the temporal 
domain 402 to locations 410a marked in the 3D spatial domain 400. The timeline grid 430 is 
5 rendered in the visual representation 18 as an overlay in front of the 2D or 3D reference surface 
404. The timeline chart 430 can be a rectangular region containing a regular or logarithmic time 
scale upon which event representations 410b are laid out. The chart 430 is arranged so that one 
dimension 432 is time and the other is location 434 based on the position of the locations 41 0a 
on the reference surface 404. As the reference surface 404 is navigated or manipulated the 
10 timelines 422 in the chart 430 move to follow the new relative location 410a positions. This 
linked location and temporal scrolling has the advantage that it is easy to make temporal 
comparisons between events since time is represented in a flat chart 430 space. The position 
410b of the event can always be traced by following the timeline 422 down to the reference 
surface 404 to the location 410a. 

15 

Referring to Figures 1 1 and 12, the TimeChart 430 can be rendered in 2 orientations, one 
vertical and one horizontal. In the vertical mode of Figure 1 1, the TimeChart 430 has the 
location dimension 434 shown horizontally, the time dimension 432 vertically, and the timelines 
422 connect vertically to the reference surface 404. In the horizontal mode of Figure 12, the 

20 TimeChart 430 has the location dimension 434 shown vertically, the time dimension 432 shown 
horizontally and the timelines 422 connect to the reference surface 404 horizontally. In both 
cases the TimeChart 430 position in the visualization representation 18 can be moved anywhere 
on the screen of die VI 202 (see Figure 1), so that the chart 430 may be on either side of die 
reference surface 404 or in front of the reference surface 404. In addition, the temporal directions 

25 of past 902 and future 904 can be swapped on either side of the focus 900. 

Interaction Interface Descriptions 

Referring to Figures 3 and 13, several interactive controls 306 support navigation and 
analysis of information within the visualization representation 1 2, as monitored by the 
30 visualization mango- 300 in connection with user events 109. Examples of the controls 306 are 
such as but not limited to a time slider 910, an instant of focus selector 912, a past time range 
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selector 914, and a future time selector 916. It is recognized that these controls 306 can be 
represented on the VI 202 (see Figure 1) as visual based controls, text controls, and/or a 
combination thereof. 

Time and Range Slider 901 

5 The timeline slider 910 is a linear time scale that is visible underneath the visualization 

representation 18 (including the temporal 402 and spatial 400 domains). The control 910 
contains sub controls/selectors that allow control of three independent temporal parameters: the 
Instant of Focus, the Past Range of Time and the Future Range of Time. 

10 Instant of focus 

The instant of focus selector 912 is the primary temporal control. It is adjusted by 
dragging it left or right with the mouse pointer across the time slider 910 to the desired position. 
As it is dragged, the Past and Future ranges move with it The instant of focus 900 (s ee Figure 
12) (also known as the browse time) is the moment in time represented at the reference surface 

15 404 in the spatial-temporal visualization representation 18. As the instant of focus selector 912 is 
moved by the user forward or back in time along the slider 910, the visualization representation 
18 displayed on die interface 202 (see Figure 1) updates the various associated visual elements of 
the temporal 402 and spatial 400 domains to reflect die new time settings. For example, 
placement of Event visual elements 410 animate along die timelines 422 and Entity visual 

20 elements 410 move along the reference surface 404 interpolating between known locations visual 
elements 410 (see figures 6 and 7). Examples of movement are given with reference to Figures 
14, 15, and 16 below. 

Past Time Rflnp ft 

25 The Past Time Range selector 914 sets the range of time before the moment of interest 

900 (see Figure 1 1) for which events will be shown. The Past Time range is adjusted by dragging 
die selector 914 left and right with the mouse pointer. The range between the moment of interest 
900 and the Past time limit can be highlighted in red (or other colour codings) on the time slider 
910. As the Past Time Range is adjusted, viewing parameters of die spatial-temporal 

30 visualization representation 18 update to reflect die change in die time settings. 
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Future Time Range 

The Future Time Range selector 914 sets the range of time after die moment of interest 
900 for which events will be shown. The Future Time range is adjusted by dragging die selector 
5 916 left and right with the mouse pointer. The range between the moment of interest 900 and the 
Future time limit is highlighted in blue (or other colour codings) on the time slider 9 10. As the 
Future Time Range is adjusted, viewing parameters of the spatial-temporal visualization 
representation 18 update to reflect the change in the time settings. 

10 The time range visible in the time scale of the time slider 910 can be expanded or 

contracted to show a time span from centuries to seconds. Clicking and dragging on the time 
slider 910 anywhere except the three selectors 912, 914, 916 will allow the entire time scale to 
slide to translate in time to a point further in the future or past Other controls 9 1 8 associated 
with the time slider 910 can be such as a "Fit 1 * button 918 for automatically adjusting the time 

15 scale to fit the range of time covered by the currently active data set displayed in the 

visualization representation 18. A scale control 918 includes a Fit control 919, a scale-expand- 
contract controls 920, a step control 923, and a play control 922, which allow the user to expand 
or contract the time scale. A step control 918 increments the instant of focus 900 forward or 
back. The* 'playback" button 920 causes the instant of focus 900 to animate forward by a user- 

20 adjustable rate. This "playback" causes the visualization representation 18 as displayed to 
animate in sync with the time slider 910. 

Association Analysis Tools 

Referring to Figures 1 and 3, association analysis functions 307 have been developed that 
take advantage of the association-based connections between Events, Entities and Locations. 

25 These functions 307 are used to find groups of connected objects 14 during analysis. Hie 

associations 16 connect these basic objects 20, 22, 24 into complex groups 27 (see Figures 6 and 
7) representing actual occurrences. The functions 307 are used to follow die associations 16 from 
object 14 to object 14 to reveal connections between objects 14 that are not immediately 
apparent Association analysis functions 307 are especially useful in analysis of large data sets 

30 where an efficient method to find and/or filter connected groups is desirable. For example, an 
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Entity 24 maybe be involved in events 20 in a dozen places/locations 22, and each of those 
events 20 may involve other Entities 24. The association analysis function 307 can be used to 
display only those locations 22 on the visualization representation 1 8 that the entity 24 has 
visited or entities 24 that have been contacted. 

5 

The analysis functions 307 provide the user with different types of link analysis, such as but 
limited to: 

1. Expanding Search 

The expanding search function 307 allows the user to start with a selected objects) 14 
1 0 and then incrementally show objects 14 that are associated with it by increasing degrees 

of separation. The user selects an object 14 or group of objects 14 of focus and clicks on 
the Expanding search button 920- this causes everything in the visualization 
representation 18 to disappear except the selected items. The user then increments the 
search depth and objects 14 connected by the specified depth are made visible the 
IS display. In this way, sets of connected objects 14 are revealed as displayed using the 

visual elements 410 and 412. 

2. Connection Search 

The Connection Search function 307 allows the user to connect any two objects 14 by 
20 their web of associations 26. The user selects any two objects 1 4 and clicks on a 

Connection Search tool (not shown). The connection search function 307 works by 
automatically scanning the extents of the web of associations 26 starting from one of the 
objects 14. The search will continue until the second object 14 is found as one of die 
connected objects 14 or until there are no more connected objects 14. If a path of 
25 associated objects 14 between the target objects 14 exists, all of the objects 14 along that 

path are displayed and die depth is automatically displayed showing die minimum 
number of links between the objects 14. 

It is recognized that die functions 307 can be used to imp lement filtering via such as but 
30 not limited to criteria matching, algorithmic methods and/or manual selection of objects 14 and 
associations 16 using the analytical properties of the tool 12. This filtering can be used to 
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highlight/hide/show (exclusively) selected objects 14 and associations 16 as represented on the 
visual representation 1 8. The Amotions 307 are used to create a group (subset) of die objects 14 
and associations 16 as desired by the user through the specified criteria matching, algorithmic 
methods and/or manual selection. Further, it is recognized that die selected group of objects 14 
5 and associations 16 could be assigned a specific name which is stored in the table 122. 

Operation of Visual Tool to Generate Visualization Representation 

Referring to Figure 14, example operation 1400 shows communications 1402 and 
movement events 1404 (connection visual elements 412 - see Figures 6 and 7) between Entities 

10 "X" and "Y" over time on the visualization representation 18. This Figure 14 shows a static 
view of Entity X making three phone call communications 1402 to Entity Y from 3 different 
locations 410a at three different times. Further, the movement events 1404 are shown on the 
visualization representation 1 8 indicating that the entity X was at three different locations 410a 
(location A3»C), which each have associated timelines 422. The timelines 422 indicate by the 

1 5 relative distance (between the elements 410b and 41 Oa) of the events (El ,E2JB3) from the instant 
of focus 900 of the reference surface 404 that these communications 1404 occurred at different 
times in the time dimension 432 of the temporal domain 402. Arrows on the communications 
1402 indicate the direction of the communications 1402, i.e. from entity X to entity Y. Entity Y 
is shown as remaining at one location 410a (D) and receiving the communications 1402 at the 

20 different times on the same timeline 422. 

Referring to Figure 15, example operation 1500 for shows Events 140b occurring within 
a process diagram space domain 400 over the time dimension 432 on the reference surface 404. 
25 The spatial domain 400 represents nodes 1 502 of a process. This Figure 1 4 shows how a 

flowchart or other graphic process can be used as a spatial context for analysis. In this case, the 
object (entity) X has been tracked through the production process to the final stage, such that the 
movements 1504 represent spatial connection elements 412 (see Figures 6 and 7). 

30 Referring to Figures 3 and 19, operation 800 of the tool 12 begins by the manager 300 

assembling 802 die group of objects 14 from the tables 1 22 via the data manager 114. The 
selected objects 14 are combined 804 via the associations 16, including assigning the connection 
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visual element 412 (see Figures 6 and 7) for the visual representation 18 between selected paired 
visual elements 410 corresponding to the selected correspondingly paired data elements 14 of the 
group. The connection visual element 412 represents a distributed association 16 in at least one 
of the domains 400, 402 between the two or more paired visual elements 410. For example, the 
5 connection element 412 can represent movement of the entity object 24 between locations 22 of 
interest on the reference surface 404, communications (money transfer, telephone call, email, 
etc. . .) between entities 24 different locations 22 on the reference surface 404 or between entities 
24 at the same location 22, or relationships (e.g. personal, organizational) between entities 24 at 
the same or different locations 22. 

10 

Next, the manager 300 uses the visualization components 308 (e.g. sprites) to generate 
806 the spatial domain 400 of the visual representation 18 to couple the visual elements 410 and 
412 in the spatial reference frame at various respective locations 22 of interest of the reference 
surface 404. The manager 300 then uses the appropriate visualization components 308 to 

IS generate 808 the temporal domain 402 in the visual representation 1 8 to include various 

timelines 422 associated with each of the locations 22 of interest, such that the timelines 422 all 
follow the common temporal reference frame. The manager 1 12 then takes the input of all visual 
elements 410, 412 from the components 308 and renders than 810to the display of the user 
interface 202. Hie manager 1 12 is also responsible for receiving 812 feedback from the user via 

20 user events 1 09 as described above and then coordinating 8 14 with the manager 300 and 

components 308 to change existing and/or create (via steps 806, 808) new visual elements 410, 
412 to correspond to the user events 109. The modified/new visual elements 410, 412 are then 
rendered to the display at step 810. 

25 Referring to Figure 16, an example operation 1600 shows animating entity X movement 

between events (Event 1 and Event 2) during time slider 901 interactions via the selector 912. 
First, the Entity X is observed at Location A at time t As the slider selector 912 is moved to the 
right, at time t+1 the Entity X is shown moving between known locations (Eventl and Event2). 
It should be noted that the focus 900 of the reference surface 404 changes such that the events 1 

30 and 2 move along their respective timelines 422, such that Event 1 moves from the future into 
the past of the temporal domain 402 (from above to below the reference surface 404). The 
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length of the timeline 422 for E vent 2 (between the Event 2 and the location B on the reference 
surface 404 decreases accordingly. As the slider selector 91 2 is moved further to the right, at 
time t+2, Entity X is rendered at Event2 (Location B). It should be noted that the Event 1 has 
moved along its respective timeline 422 further into the past of the temporal domain 402, and 
S event 2 has moved accordingly from the future into the past of the temporal domain 402 (from 
above to below the reference surface 404), since the representation of the events 1 and 2 are 
linked in the temporal domain 402. Likewise, the entity X is linked spatially in the spatial 
domain 400 between event 1 at location A and event 2 at location B. It is also noted that the 
Time Slider selector 912 could be dragged along the time slider 910 by die user to replay the 
10 sequence of events from time t to t+2, or from t+2 to t, as desired. 

Referring to Figure 17, the visual reresentation IS shows connection visual elements 412 
between visual elements 410 situated on selected various timelines 422. The timelines 422 are 
coupled to various locations 22 of interest on the geographical reference frame 404. In this case, 
IS the elements 412 represent geographical movement between various locations 22 by entity 24, 
such that all travel happened at some time in the future with respect to the instant of focus 
represented by the reference plane 404. 

Referring to Figure 18, the spatial domain 400 is shown as a geographical relief map. 

20 The timechart 430 is superimposed over the spatial domain of the visual representation 1 8, and 
shows a time period spanning from December 3 rd to January 1 st for various events 20 and entities 
24 situated along various timelines 422 coupled to selected locations 22 of interest It is noted 
that in this case the user can use the presented visual representation to coordinate the assignment 
of various connection elements 412 to the visual elements 410 (see Figure 6) of the objects 20, 

25 22, 24 via the user interface 202 (see Figure 1), based on analysis of the displayed visual 

representation 1 8 content. A time selection 950 is January 30, such that events 20 and entities 
24 within the selection box can be further analysed. It is recognised that the time selection 950 
could be used to represent the instant of focus 900 (see Figure 9). 

30 It will be appreciated that variations of some elements are possible to adapt the invention 

for specific conditions or functions. The concepts of the present invention can be further 
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extended to a variety of other applications that are clearly within the scope of this invention. 
Having thus described the present invention with respect to preferred embodiments as 
implemented, it will be apparent to those skilled in the art that many modifications and 
enhancements are possible to the present invention without departing from the basic concepts as 
described in the preferred embodiment of the present invention. Therefore, what is intended to 
be protected by way of letters patent should be limited only by the scope of the following claims. 
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WE CLAIM: 

1 . A method for creating a multidimensional visual representation of a group of data 
elements having integrated temporal and spatial properties, the data elements being included in 
the visual representation as corresponding visual elements, the data elements of the group linked 

5 by at least one association, the method comprising the steps of: 

assembling the group of data elements using the at least one association; 
generating a spatial domain of the visual representation to include a reference surface for 
providing a spatial reference frame having at least two spatial dimensions, die reference surface 
for relating a first visual element representing a first data element of the group to a first location 
10 of interest in the spatial reference frame and relating a second visual element representing a 
second data element of die group to a second location of interest in the spatial reference frame; 

generating a temporal domain of the visual representation operatively coupled to the 
spatial domain, the temporal domain for providing a common temporal reference frame for the 
locations of interest, the temporal domain including a first time track coupled to the first location 
IS of interest and a second time track coupled to die second location of interest, the first visual 

element positioned on the first time track and die second visual element positioned on die second 
time track, each of the time tracks configured for visually representing a respective temporal 
sequence of a plurality of the data elements at each of die locations of interest of the reference 
surface; and 

20 assigning a connection visual element in the visual representation between the first visual 

element and the second visual element, the connection visual element for representing a 
distributed association in at least one of the domains between the first visual element and the 
second visual element; 

wherein the visual representation is displayed on a user interface for subsequent 

25 interaction with user events. 

2. The method of claim 1, wherein die visual representation of the group of data elements is 
selected from the group comprising; a concurrent time ami geographic contact, and a concurrent 
time and diagrammatic context 

30 
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3. The method of claim 2 further comprising the step of configuring the reference surface 
for providing an instant of focus in the temporal reference frame for at least some of the temporal 
and spatial properties of the group of data elements, the instant of focus coupled to the locations 
of interest of the reference surface, the instant of focus selected by the user events. 

5 

4. The method of claim 3 further comprising the step of configuring the temporal reference 
frame for providing a time range selected from the group comprising; a past range of the 
temporal sequence preceding the instant of focus, and a future range of the temporal sequence 
after the instant of focus. 

10 

5. The method of claim 4 further comprising the step of intersecting the first time track 
through the first location of interest such that the past range of the first time track extends from 
one side of the reference surface and the future range of the first time track extends from the 
other side of the reference surface, the instant of focus located on the first time track at the 

1 S intersection point between the first time track and the reference surface. 

6. The method of claim 5 further comprising the step of arranging a first plurality of the 
visual elements along the first time track according to the times at which the visual elements 
occurred in the temporal reference frame. 

20 

7. The method of claim 6 further comprising the step of intersecting the second time track 
through the second location of interest such that the past range of the second time track extends 
from one side of the reference surface and the future range of the first time trade extends from 
the other side of the reference surface, the second instant of focus located on the second time 

25 track at file intersection point between the second time track and die reference surface. 

8. The method of claim 7 further comprising the step of arranging a second plurality of the 
visual elements along die second time track according to die times at which die visual elements 
occurred in the temporal reference frame. 

30 
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9. The method of claim 6, wherein the location in the temporal reference frame of each of 
the plurality of visual elements on the time tracks is proportional to the distance from the instant 
of focus associated with the reference surface. 

5 10. The method of claim 9, wherein the time tracks are represented as timelines in the visual 
representation. 

1 1 . The method of claim 10, wherein die temporal reference frame has a scale selected from 
the group comprising linear and logarithmic. 

10 

1 2. The method of claim 2 further comprising the step of maintaining an orientation of the 
time tracks with respect to the reference surface such that changes in the attitude of the reference 
surface in response to the user events results in a corresponding change in the orientation of the 
time tracks. 

15 

13. The method of claim 12, wherein the orientation angle between the time tracks is 90 
degrees. 

14. The method of claim 2 further comprising the step of maintaining an orientation of the 
20 time tracks with respect to the reference surface such that changes in the attitude of the reference 

surface in response to the user events does not result in a corresponding change in the orientation 
of the time tracks. 

15. The method of claim 14, wherein the orientation of the time tracks is such that the length 
25 of the time tracks is maximized as perceived by a user of the user interface. 

16. The method of claim 2 further comprising die step of overlapping a time chart on the first 
time track and the second time track, the time chart having a time axis and a spatial axis for 
representing the temporal reference frame and at least one of the spatial dimensio ns respectively. 

30 
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17. The method of claim 16 further comprising the step of maintaining an orientation of die 
time chart with respect to the reference surface such that changes in the attitude of the reference 
surface in response to die user events does not result in a corresponding change in the orientation 
of die time chart. 

5 

18. The method of claim 17, wherein the time chart is represented as a rectangular region. 

19. The method of claim 16 further comprising the step of configuring the time chart for 
providing an instant of focus in the temporal reference frame for at least some of the temporal 

10 and spatial properties of the group of date elements, the instant of focus coupled to the locations 
of interest of the reference surface, the instant of focus selected by the user events. 

20. The method of claim 19 further comprising the step of configuring the time chart for 
providing a time range selected from the group comprising; a past range of the temporal 

IS sequence proceeding die instant of focus, and a future range of the temporal sequence after the 
instant of focus. 

21. The method of claim 20, wherein the time chart has a superimposed grid for representing 
the time axis and the spatial axis. 

20 

22. The method of claim 2, wherein types of the data elements are selected from the group 
comprising; entity, location, and event 

23. The method of claim 22, wherein the event data element type represents an action taking 
25 place at a particular one of the locations of interest in the spatial reference frame and at a 

particular time in the temporal reference frame. 

24. The method of claim 23, wherein the event data element type has data properties and 
display properties selected from the group comprising; a short text label, description, location, 

30 start-time, end-time, general event type, icon reference, visual layer settings, priority, status, user 
comment, certainty value, source of information, and default + user-set color. 
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25. The method of claim 22, wherein the entity data element type represents an actor 
involved in a selected event 

5 26. The method of claim 25, wherein the entity data element type has data properties and 
display properties selected from the group comprising; short text label, description, general entity 
type, icon reference, visual layer settings, priority, status, user comment, certainty value, source 
of information, and default + user-set color. 

10 27. The method of claim 22, wherein the location data element type represents a location 
within the spatial reference frame. 

28. The method of claim 27, wherein the location data element type has data properties and 
display properties selected from the group comprising; position coordinates, a label, description, 

15 color information, precision information, location type, non-geospatial flag and user comments. 

29. The method of claim 28, wherein the location data element type is selected from the 
group comprising; a physical location on a geospatial map, a physical location as a node in a 
diagram, and a virtual location related to a geospatial map. 

20 

30. The method of claim 22, wherein the at least one association describes a pairing between 
two or more of the data elements. 

31 . The method of claim 30, wherein the connection visual element is a solid line 

25 representing a direct connection between the first visual element and the second visual element. 

32. The method of claim 31, wherein the solid line has a pointer for indicating a vector 
property of the visual connection element 
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33. The method of claim 31, wherein the connection visual element is a trail represented as a 
series of connected lines tracing known locations in the spatial reference frame of a selected 
entity visual element over time represented by the temporal reference frame. 

5 34. The method of claim 2 further comprising the step of updating the visual elements in the 
visual representation in response to the user events. 

35. The method of claim 34, wherein the user events are generated in response to 
manipulation by the user of an interactive control for modifying visual properties of the visual 

10 representation. 

36. The method of claim 35, wherein the interactive control is selected from the group 
comprising a time range selector and an instant of focus selector. 

1 5 37. The method of claim 35 further comprising the step of animating the display of at least 
one of die visual elements of the visual representation in response to the manipulation of the 
interactive control 

38. The method of claim 37, wherein the display of die connection visual element of the 
20 visual representation is animated in response to the manipulation of the interactive controls. 

39. The method of claim 38, wherein die connection visual element is coupled to the 
movement of an entity visual element across the visual representation between the first location 
of interest and the second location of interest, entity visual element representing an actor 

25 involved in a selected event. 



40. The method of claim 35 further comprising the step of applying a filtering function to the 
visual elements and the at least one related association to select a subgroup thereof. 
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41. The method of claim 40 further comprising the step of selecting the subgroup according 
to a method selected from the group comprising; criteria matching, algorithmic methods, and 
manual selection. 

5 42. The method of claim 40 further comprising the step of processing the selected subgroup 
of visual elements to change the presentation of the visual representation selected from the group 
comprising; highlighting the subgroup and removing the subgroup. 

43. A system for creating a multidimensional visual representation of a group of data 

10 elements having integrated temporal and spatial properties, the data elements being included in 
the visual representation as corresponding visual elements, the data elements of the group linked 
by at least one association, the system comprising: 

a visualization manager for assembling the group of data elements using the at least one 
association and for assigning a connection visual element in the visual representation between a 

IS first visual element representing a first data element of the group and a second visual element 
representing a second data element of the group; 

a spatial visualization component configured for generating a spatial domain of the visual 
representation to include a reference surface for providing a spatial reference frame having at 
least two spatial dimensions, the reference surface for relating the first visual element to a first 

20 location of interest in the spatial reference frame and relating the second visual element to a 
second location of interest in the spatial reference frame; and 

a temporal visualization component configured for generating a temporal domain of the 
visual representation operatively coupled to the spatial domain, the temporal domain for 
providing a common temporal reference frame for the locations of interest, the temporal domain 

25 including a first time track coupled to the first location of interest and a second time track 
coupled to the second location of interest, the first visual element positioned on the first time 
track and the second visual element positioned on the second time track, each of the time tracks 
configured for visually representing a respective temporal sequence of a plurality of the data 
elements at each of the locations of interest of the reference surface; and 
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wherein the connection visual element represents a distributed association in at least one 
of the domains between the first visual element and the second visual element such that the 
visual representation is displayed on a user interface for subsequent interaction with user events. 

5 44. A computer program product for creating a multidimensional visual representation of a 
group of data elements having integrated temporal and spatial properties, the data elements being 
included in the visual representation as corresponding visual elements, the data elements of the 
group linked by at least one association, the computer program product comprising: 
a computer readable medium; 

10 a visualization module stored on the computer readable medium for assembling the group 

of data elements using the at least one association and for assigning a connection visual element 
in the visual representation between a first visual element representing a first data element of the 
group and a second visual element representing a second data element of the group; 

a spatial visualization module stored on the computer readable medium for generating a 

1 S spatial domain of the visual representation to include a reference surface for providing a spatial 
reference frame having at least two spatial dimensions, the reference surface for relating the first 
visual element to a first location of interest in the spatial reference frame and relating the second 
visual element to a second location of interest in the spatial reference frame; and 

a temporal visualization module stored on the computer readable medium for generating 

20 a temporal domain of the visual representation operatively coupled to the spatial domain, the 
temporal domain for providing a common temporal reference frame for th e locations of interest, 
the temporal domain including a first time track coupled to the first location of interest and a 
second time track coupled to the second location of interest, the first visual element positioned 
on the first time track and die second visual element positioned on the second time trade, each of 

25 the time tracks configured for visually representing a respective temporal sequence of a plurality 
of the data elements at each of the locations of interest of the reference surface; and 

wherein the connection visual element represents a distributed association in at least one 
of the domains between the first visual element and the second visual element such that the 
visual representation is displayed on a user interface for subsequent interaction with user events. 

30 
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